Sample Complexity of Policy Gradient Finding Second-Order Stationary Points

The policy-based reinforcement learning (RL) can be considered as maximization of its objective. However, due to the inherent non-concavity objective, policy gradient method a first-order stationary point (FOSP) cannot guar- antee maximal point. A FOSP minimal or even saddle point, which is undesirable for RL. It has found that if all points are strict, second-order station- ary (SOSP) exactly equivalent local maxima. Instead FOSP, we consider SOSP convergence criteria characterize sample complexity gradient. Our result shows converges an (ε, √εχ)-SOSP with probability at least 1 − O(δ) after total cost O(ε−9/2)sinificantly improves state art O(ε−9).Our analysis based on key idea decomposes parameter space Rp into three non-intersected regions: non-stationary region, and optimal then making improvement objective RL in each region. This technique potentially generalized extensive methods. For complete proof, please refer https://arxiv.org/pdf/2012.01491.pdf.